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Conformance  Testing  Service 


- ABSTRACT  - 


NIST  has  been  tasked  by  the  CALS  Project  Office  to  organize  an  SGML  Conformance  Testing  Service. 
This  document  establishes  operating  policy  and  procedures  for  the  Computer  Systems  Laboratory’s  (CSL) 
testing  program  for  Federal  Information  Processing  Standards  (FIPS)  152,  Standard  Generalized  Markup 
Language  (SGML),  parsers.  The  testing  methodology  is  based  on  ANSI  X3. 190- 1992,  Text  and  Office 
Systems  - Conformance  Testing  for  Standard  Generalized  Markup  Language  Systems.  This  document 
contains  operating  policy  for  an  SGML  Conformance  Testing  Service  and  is  not  intended  to  explain  the 
detailed  procedures  that  can  be  found  in  the  documentation  associated  with  the  SGML  parser  validation 
system,  commonly  called  the  SGML  Test  Suite. 
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1.  INTRODUCTION 


1.1  Purpose 

This  document  establishes  operating  policy  and  procedures  for  the  Computer  Systems  Laboratory’s 
(CSL)  validation  program  for  Federal  Information  Processing  Standards  (FTPS)  152,  Standard 
Generalized  Markup  Language  (SGML)  [1],  parsers.  The  testing  methodology  is  based  on  ANSI 
X3. 190- 1992,  Text  and  Office  Systems  - Conformance  Testing  for  Standard  Generalized  Markup 
Language  Systems  [2].  This  document  contains  operating  policy  for  an  Standard  Generalized 
Markup  Language  [3]  Conformance  Testing  Service  and  is  not  intended  to  explain  the  detailed 
procedures  that  can  be  found  in  the  documentation  associated  with  the  SGML  parser  validation 
system,  commonly  called  the  SGML  Test  Suite. 


1.2  Background 

The  Computer  Systems  Laboratory  of  the  National  Institute  of  Standards  and  Technology  (NIST) 
is  responsible  for  establishing  conformance  testing  programs  for  Federal  Information  Processing 
Standards  (FTPS).  In  carrying  out  this  responsibility,  CSL  specifies  the  necessary  conformance  test 
specifications,  test  methods  (i.e.,  test  suites,  test  tools,  and  technical  procedures),  validation 
procedures,  and  testing  laboratories  for  testing  product  compliance  to  FTPS. 

The  CSL  conformance  testing  program  for  SGML  provides  for  the  testing  of  SGML  parsers  for 
conformance  to  the  FTPS.  Testing  of  SGML  parsers  to  determine  the  degree  to  which  they  conform 
to  the  Federal  Standards  may  be  required  by  Government  departments  and  agencies  in  accordance 
with  the  FTPS,  the  Federal  Information  Resources  Management  Regulation  (FIRMR)  201-20.303, 
201-20.304,  and  201-39.1002,  and  the  associated  Federal  ADP  and  Telecommunications  Standards 
Index  (hereafter  referred  to  as  the  Index).  The  testing  information  available  from  CSL  aids  Federal 
procurement  authorities,  as  well  as  industry,  in  determining  if  the  SGML  parsers  offered  to  the 
government  comply  with  the  Government’s  requirements  for  FTPS. 

The  testing  information  provided  by  the  SGML  Conformance  Testing  Service  is  published  in 
Certificates  of  Validation,  Validation  Summary  Reports  (VSR),  Registered  Validation  Summary 
Reports,  The  Validated  Products  List  (VPL),  and  validation  procedures. 


1.3  Validated  Products  List  (VPL) 

CSL  publishes,  on  a quarterly  basis,  a Validated  Products  List  that  is  a collection  of  registers 
describing  implementations  of  Federal  Information  Processing  Standards  that  have  been  validated 
for  conformance  to  FIPS.  The  VPL  contains  information  about  the  organizations,  test  methods,  and 
procedures  that  support  the  validation  programs  for  the  FIPS  identified  in  this  document.  For 
example,  the  VPL  contains  information  about  computer  security,  programming  languages,  SQL, 
Portable  Operating  System  Interface  for  Computer  Environments  (POSIX),  Government  Open 
Systems  Interconnection  Profile  (GOSTP),  etc.  When  appropriate,  SGML  will  be  added  to  the  VPL. 
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For  SGML,  the  VPL  will  identify  the  SGML  parsers  and  the  computer  system  environment  on 
which  the  validation  testing  took  place. 

The  VPL  is  intended  to  serve  as  an  index  to  more  detailed  information.  Each  SGML  entry  in  the 
VPL  will  be  a very  limited  extract  from  a VSR  available  from  CSL.  The  VSR  will  also  be 
available  from  the  testing  laboratories. 


1.4  SGML  Validation  Services 

The  SGML  Validation  Service  will  use  testing  laboratories  to  provide  a testing  service  to  evaluate 
conformance  of  SGML  parsers  to  requirements  of  FIPS  152  (FTPS  SGML). 

Prior  to  validation,  the  SGML  Test  Suite  is  purchased  by  the  SGML  supplier  (vendor)  from  CSL. 
The  SGML  supplier  performs  self-testing  to  identify  and  correct  nonconformities.  A formal 
validation,  at  a latter  date,  consists  of  an  audited  execution  of  the  same  tests  cases.  The  format  of 
the  validation  is  a live  test  demonstration  by  the  supplier’s  staff  at  the  supplier’s  site  or  at  the 
testing  laboratory.  The  testing  laboratory  staff  must  review  and  approve  the  environment  and 
proposed  method  of  execution  for  the  live  test  demonstration.  The  testing  laboratory  staff  then 
audits  the  live  test  demonstration  and  documents  the  results  in  a VSR.  The  timing  of  the  validation 
is  controlled  by  a series  of  rigorous  procedures  detailed  within  this  manual.  The  SGML  supplier 
must  provide  technical  support  personnel  for  the  validation  service. 

The  SGML  Test  Suite  is  a package  containing  over  one  thousand  test  scripts.  In  order  for  an 
SGML  parser  to  conform  to  FIPS  152,  the  SGML  parser  must  be  validated  by  passing  all  of  these 
test  scripts. 


1.5  Scope  of  SGML  Validation 

SGML  validation  has  a very  narrow  scope.  The  results  of  validation  apply  only  to  the  SGML 
parser  tested,  its  operating  environment,  and  the  SGML  Test  Suite  version  used.  The  Certificate 
of  Validation  verifies  that  the  product  has  been  tested  using  the  Official  National  Institute  of 
Standards  and  Technology  SGML  Conformance  Test  Suite  for  the  Federal  Information 
Processing  Standards  Publication  152  and  that  the  test  results  obtained  have  been  validated 
by  NIST.  Additional  features  of  the  SGML  product  are  not  being  tested.  Therefore,  a 
successfully  validated  product  that  receives  a certificate  only  implies  that  the  SGML  parser 
within  the  product  passed  aU  the  tests  thereby  demonstrating  its  conformance  with  FIPS  152. 

The  output  from  running  the  test  scripts  must  be  compliant  with  ANSI  X3. 190-1992,  Text  and 
Ojfice  Systems  - Conformance  Testing  for  Standard  Generalized  Markup  Language  (SGML) 
Systems.  The  SGML  product  must  output  the  parser  results  from  running  the  test  scripts  in  a 
Reference  Application  for  SGML  Testing  (RAST)  format.  RAST  interprets  all  markup  to  allow  for 
machine  comparison  of  test  results  for  documents  conforming  to  ISO  8879.  RAST  indicates  in  a 
standard  way  when  a tag,  processing  instruction,  or  data  is  recognized  by  the  parser,  replacing 
references  and  processing  markup  declarations  and  marked  sections  appropriately. 
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The  results  should  be  reproducible  for  the  environment  tested,  but  not  necessarily  for  other 
environments.  Therefore,  it  is  important  to  document  accurately  the  significant  software  and 
hardware  components  involved. 


1.5.1  Identification  of  Environment  Tested 

The  supplier  of  an  SGML  parser  will  identify  the  SGML  parser  (and  components),  the  operating 
system,  and  the  hardware  systems  to  be  used  in  a given  validation.  Version  and  release  numbers, 
as  well  as  model  numbers,  will  be  identified  as  appropriate.  All  of  this  information  is  provided  by 
the  supplier  on  the  validation  form  "SGML  Validation  Request  Form"  (See  Appendix  B).  This 
information  is  confirmed,  to  the  extent  possible,  at  the  time  of  the  validation.  The  VSR  written  by 
the  testing  service  will  additionally  identify  the  SGML  Test  Suite  version  number  and  the  interfaces, 
features,  or  profiles  tested  as  well  as  the  test  results.  The  VSR  will  document  configuration  options, 
parameters,  and  any  special  procedures  used  in  validation  testing. 


1.5.2  Caveats 

Validation  testing  does  not  warrant  that  the  product  tested  will  perform  according  to  all  the 
vendor’s  assertions.  Validation  only  specifies  that  the  parser  within  the  vendor’s  product  was 
tested  and  that  the  test  results  passed  the  requirements  of  FIPS  152. 

Validation  testing  does  not  warrant  that  the  product  tested  is  free  of  nonconformities,  even 
if  all  tests  are  passed.  The  practical  goal  of  SGML  validation  is  to  identify  SGML  parsers  which 
may  be  procured  (by  federal  agencies)  and  used  to  develop  application  programs  which  meet  the 
FTPS  goals  of  portability  and  interoperability. 

The  SGML  Test  Suite  is  not  designed  to  replace  the  vendor’s  quality  assurance  testing  or 
systematically  to  detect  inconsistencies  or  "bugs."  The  primary  technical  goal  of  SGML 
validation  testing  is  to  verify  that  the  SGML  product  correctly  supports  all  required  features  of  FIPS 
152.  Rather  than  exhaustive  testing  of  permutations  of  features,  the  SGML  Test  Suite  contains  a 
carefully-chosen  set  of  test  cases  which  cover  the  required  syntax  and  demonstrate  the  correct 
implementation  of  each  of  the  applicable  general  rules  of  the  standard. 

Validation  is  not  intended  as  a means  of  performance  benchmarking.  The  VSR  does  not 
contain  information  about  the  speed,  cost,  or  efficiency  of  executing  the  validation  programs. 

Validation  is  not  intended  to  evaluate  all  optional  features  of  the  standard.  For  the  SGML 
standard,  optional  features  are  allowed  but  will  not  be  tested  except  for  the  two  required  by  FTPS 
152:  OMITTAG  and  FORMAL. 


1 .6  Role  of  Procuring  Agency 

The  procuring  agency  has  the  authority  to  extrapolate  from  the  test  results  published  in  the  VPL 
to  other  hardware/software  environments,  based  on  additional  research  by  the  procuring  agency. 
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demonstrations  or  warranties  by  the  SGML  supplier,  etc.  The  procuring  agency,  not  CSL,  has  the 
responsibility  of  reviewing  the  SGML  validations  listed  in  the  VPL  and  determining  the 
applicability  of  these  validations  to  the  hardware/software  environment  involved  in  a specific 
procurement.  The  criteria  for  applicability  of  a VSR  or  Certificate  should  be  appropriate  to  the  size 
and  timing  of  the  procurement. 
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2.  REGISTRATION  OF  VALIDATION  RESULTS 


2. 1 Overview 

Validation  testing  consists  of  a third  party  reviewing  the  supplier’s  testing  procedures,  witnessing 
the  running  of  the  conformance  tests,  evaluating  the  test  results,  and  reporting  the  results  of  that 
testing  in  a VSR.  If  the  validation  procedures  are  followed  and  the  VSR  shows  that  the  tested 
product  demonstrated  conformance  to  the  standard  (as  defined  by  the  test  method  and  these 
validation  procedures),  a Certificate  of  Validation  and  Registered  VSR  is  issued  for  the  SGML 
parser  on  the  computer  and  operating  system  tested. 

A Certificate  of  Validation  is  issued  if  all  required  tests  are  passed  and  if  all  the  criteria  in  Section 
2.5.1  are  met.  SGML  parsers  with  a Certificate  of  Validation  are  viewed  as  being  "validated"  at 
the  FTPS  levehoption/profile  that  testing  was  performed. 


2.2  Specification  of  Validation  Platform 

For  validation  testing,  the  SGML  parser  and  its  environment  are  specified  as  follows: 

a.  The  name  of  the  SGML  parser  (product)  and  its  version  or  release  number.  [The  version  or 
release  number  specifies,  at  a minimum,  the  number  under  which  the  product  is  identified 
for  marketing  purposes.  The  number  need  not  specify  digits  at  the  bug-fix  level,  unless  they 
are  relevant  to  SGML  conformance.] 

b.  The  name  of  the  individual  SGML  components  (and  version  or  release  numbers)  used  in  the 
validation. 

c.  The  name  of  the  operating  system  and  its  version  or  release  number. 

d.  The  name  of  the  computer  hardware,  including  series  and  model  numbers. 

All  of  this  information  is  provided  by  the  supplier  on  the  validation  form  "SGML  Validation 

Request  Form."  This  information  is  confirmed,  to  the  extent  possible,  at  the  time  of  the  validation. 

The  version  of  the  SGML  Test  Suite  and  the  date  of  the  most  recent  maintenance  change  to  the  test 

suite  are  documented. 


2.3  Validation  Summary  Report  (VSR) 

The  results  of  validation  testing  are  documented  in  a VSR.  In  addition  to  a specification  of  the 
validation  platform,  the  VSR  contains: 

a.  a table  showing  the  pass/fail  results  in  various  categories  for  each  applicable  test  suite  type, 

b.  documentation  of  changes  made  to  the  SGML  test  suite  in  order  to  install  implementation- 
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defined  SGML  language, 

c.  a description  of  test  results  for  informational  tests. 

The  information  for  the  VSR  is  based  primarily  on  test  results  obtained  by  a CSL  recognized  testing 
laboratory,  witnessing  the  execution  of  the  SGML  Test  Suite  on  the  supplier’s  SGML  parser.  The 
VSR  also  contains  information  from  statements  made  by  the  supplier  concerning  identification  of 
the  product,  SGML  parser  options,  and  the  operating  environment  in  which  validation  testing  was 
conducted. 

Each  VSR  documents  the  execution  of  the  entire  test  suite  on  at  least  one  validation  platform.  If 
appropriate,  validation  results  for  multiple  platforms  with  identical  results  (with  respect  to 
conformance  testing)  will  be  packaged  into  one  document. 


2.4  Criteria  for  Registration 

The  VSR  may  be  registered  with  CSL  if  the  following  criteria  are  met: 

a.  The  SGML  parser  has  been  tested  with  the  latest  official  release  of  the  SGML  Test  Suite. 

b.  The  validation  was  performed  by  a CSL  recognized  testing  laboratory. 

c.  All  validation  procedures  and  processing  steps  have  been  completed,  including  review  and 
release  of  the  VSR  by  the  testing  laboratory. 

d.  A final  VSR  has  been  prepared  reporting  the  results  of  validation  testing. 

e.  The  SGML  parser  is  a released  (commercially  available)  product  or  this  is  the  first  time  the 
pre-release  product  has  been  tested.  Pre-release  products  may  be  validated  only  once,  and 
revalidation  is  required  to  remove  the  "Pre-release"  designation. 

There  is  only  one  category  under  which  SGML  parsers  will  be  listed  in  the  VPL:  SGML  Parsers 

with  Certificates. 


2.5  SGML  Parsers  with  Certificates 

A CSL  Certificate  of  Validation  is  issued  if  the  SGML  parser  tested  demonstrates  conformance  to 
the  respective  FIPS  based  on  the  SGML  Test  Suite.  The  Certificate  of  Validation  identifies  the 
SGML  parser  tested,  the  version  of  the  applicable  SGML  Test  Suite,  the  FIPS  (including  applicable 
level)  tested,  the  test  suite  types  executed,  and  the  hardware/software  environment  on  which  the 
SGML  parser  was  tested. 

The  Certificate  of  Validation  is  valid  regardless  of  subsequent  SGML  Test  Suite  releases. 
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2.5.1  Certificate  of  Validation  Criteria 


A CSL  Certificate  of  Validation  is  issued  for  an  SGML  parser  if  the  SGML  parser  passed  all 
required  SGML  Test  Suite  tests  for  the  applicable  test  suite  type(s).  The  SGML  parser  must  also 
meet  the  criteria  for  registration  in  Section  2.4. 
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3.  SGML  VALIDATION  PROCEDURES 


3.1  Submitting  SGML  Parsers  for  Validation 

The  FIRMR  refers  to  the  Index  that  provides  terminology  for  agencies  to  use  when  incorporating 
FIPS  in  Government  procurements.  This  terminology  requires  that  SGML  parsers  entering  the 
Federal  Government  inventory  be  validated.  This  requirement  may  be  satisfied  by  supplying  a 
current  Certificate  of  Validation  for  the  SGML  parser,  or,  at  the  option  of  the  procuring  agency, 
temporarily  satisfy  this  requirement  by  submitting  the  SGML  parser  for  validation. 

The  phrase  "submission  for  validation"  as  used  in  the  Index  means  that  a letter  has  been  received 
by  the  test  laboratory  requesting  that  the  SGML  parser  be  validated  for  the  purposes  of  offering  the 
SGML  parser  to  the  Government.  When  such  a request  is  received,  the  test  laboratory  will  send 
the  requestor  a letter  acknowledging  receipt  of  the  request  and  indicate  the  month  in  which 
validation  testing  is  scheduled  to  take  place.  This  letter  may  be  offered  to  Government  departments 
and  agencies  as  proof  of  submission  for  validation. 

The  SGML  parser  remains  submitted  for  validation  so  long  as  all  information  necessary  to  complete 
the  validation  process  is  provided  before  the  end  of  the  month  previous  to  the  scheduled  validation. 

If  the  information  is  not  provided  by  the  end  of  the  month  previous  to  the  scheduled  validation,  and 
if  an  alternate  schedule  has  not  been  specified  in  writing  by  the  test  laboratory,  then  the  requestor 
will  no  longer  be  considered  scheduled.  To  be  rescheduled  the  requestor  must  submit  a letter 
requesting  a new  validation  date. 


3.2  Requests  for  Validation 

A request  for  validation  services  shall  be  in  the  form  of  a letter  to  any  of  the  approved  test 
laboratories.  NIST  may  be  contacted  at  the  following  address  to  receive  a list  of  the  approved  test 
laboratories: 

National  Institute  of  Standards  and  Technology  (NIST) 

Computer  Systems  Laboratory 
Systems  and  Software  Technology  Division 
Office  Systems  Engineering  Group 
Building  225,  Room  B266 
Gaithersburg,  Maryland  20899  (U.S.A.) 

Ronald  Wilson 

Telephone  (301)  975-3352  (Direct  line) 

(301)  975-3345  (Secretary) 

(301)  926-3696  (FAX) 

e-mail:  wilson@sst.ncsl.nist.gov  (INTERNET) 

The  following  information  should  be  provided: 

a.  Desired  and  alternate  month  of  testing, 

b.  SGML  parser  identification, 

c.  Point  of  contact  for  validation,  and 
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d.  Statement  specifying  whether  or  not  the  request  is  submitted  for  validation  for  the  purpose 
of  offering  the  SGML  parser  to  the  Government  in  response  to  a Request  For  Proposal 
(RFP). 

The  SGML  Validation  Request  Form  in  Appendix  B may  be  sent  instead  of  the  letter  of  request  to 
provide  the  above  information. 

A request  for  validation  of  an  SGML  parser  should  be  received  by  the  test  laboratory  no  later  than 
90  calendar  days  prior  to  the  month  desired  for  on-site  testing.  Requests  for  changes  to  an  existing 
test  date  should  be  submitted  no  later  than  30  days  prior  to  the  scheduled  month  of  testing. 
Requestors  may  choose  to  have  one  or  more  SGML  parsers  validated  at  the  same  time,  or  may 
schedule  them  separately,  according  to  their  needs. 

Upon  receipt  of  a request  for  validation,  the  test  laboratory  will  provide  prevalidation  materials 
(forms  and  lists)  to  the  requestor.  The  test  laboratory  will  also  verify  that  the  requestor  has  a 
current  license  to  use  the  SGML  Test  Suite. 

In  order  to  guarantee  timely  validation,  the  required  preliminary  prevalidation  material  must  be 
forwarded  to  the  test  laboratory  60  calendar  days  prior  to  the  month  of  the  scheduled  on-site  testing. 
Additional  prevalidation  material,  as  specified  by  the  test  laboratory,  must  be  forwarded  30  calendar 
days  prior  to  the  month  of  the  scheduled  on-site  testing.  Upon  receipt  of  the  prevalidation  material, 
an  actual  date  within  the  scheduled  month  will  be  established  for  on-site  testing.  On-site  testing 
dates  will  not  be  established  for  first  time  validations  until  after  a successful  review  of  prevalidation 
materials  by  the  test  laboratory. 

All  hardware,  software  (except  for  the  SGML  Test  Suite),  and  technical  support  must  be  provided 
by  the  requestor.  The  test  laboratory  determines  the  location  where  the  on-site  testing  will  take 
place. 


3.2.1  Typical  Validation  Schedule 

The  following  is  a typical  schedule  for  a validation: 

(Where  "T"  represents  the  day  on-site  testing  begins  and  or  ’-h’  integer  represent  the  number  of 
calendar  days  before  or  after  the  beginning  on-site  testing  day.) 

T - 90  Client  purchases  SGML  Validation  Test  Suite 

T - 90  Client  contacts  test  laboratory  to  request  validation  services 

T - 60  Client  submits  preliminary  prevalidation  materials  and  disputed  tests,  if  any 

T - 30  All  prevalidation  test  materials  received 

T - 30  Resolve  all  outstanding  issues  and  schedule  on-site  testing 

T On-site  testing  begins  (scheduled  month) 

T + 10  Draft  VSR  available 

T + 30  Final  VSR  available 
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3.3  Disputed  and  Withdrawn  Tests 


If  a supplier  believes  a test  is  in  error,  the  supplier  must  send  an  error  report  to  the  test  laboratory 
(who  will  forward  it  to  CSL)  with  associated  rationale  and  detailed  documentation  no  later  than  60 
days  before  the  on-site  testing  so  that  CSL  can  make  a determination  of  the  appropriate  action. 
CSL  will  notify  the  supplier  of  the  determination  thirty  days  prior  to  the  on-site  visit.  If  the  test 
is  determined  to  be  nonstandard,  then  it  will  be  withdrawn  from  the  SGML  Test  Suite,  or  it  may 
be  designated  as  an  informational  test  for  purposes  of  reporting  in  the  VSR. 

A Test  Method  Control  Procedures  Model  will  be  employed  in  order  to  resolve  disputes  concerning 
the  SGML  Test  Suite.  The  purpose  of  the  Test  Method  Control  Executive  Committee  (see 
Appendix  D)  is  to  resolve  issues  in  a fair  and  speedy  fashion.  The  committee  is  responsible  for: 

Resolving  challenges  to  the  test  method. 

Overseeing  the  maintenance  of  the  test  method. 

Coordinating  issues  of  interpretations  of  the  standard. 

Procedures  and  documentation  associated  with  the  test  method. 

Advising  on  the  proper  use  of  the  test  method. 

Recommending  changes  to  these  procedures. 

Accepting/recognizing  new  test  methods,  and. 

Recommending  improvements  for  the  next  revision  of  the  test  method. 

Any  technical  issues  that  are  not  resolved  between  CSL  and  the  supplier  by  the  on-site  date  may 
be  submitted  by  the  supplier  for  formal  interpretation  to  the  appropriate  Standards  Committee  (e.g., 
SC  18,  Working  Group  8)  or  to  CSL  in  accordance  with  the  latest  version  of  FTPS  PUB  29, 
"Interpretation  Procedures  for  Federal  Information  Processing  Standards  for  Software."  FTPS  PUB 
29  is  available  from  the  National  Technical  Information  Service  (NTIS),  Springfield,  VA  22161, 
telephone  number  (703)  487-4750.  If  formal  interpretation  is  requested  from  a Standards 
Committee,  CSL  must  receive  a copy  of  the  request  for  interpretation  and  all  other  relevant 
correspondence. 

If  a dispute  is  not  resolved  at  the  time  of  on-site  testing,  and  if  CSL  judges  the  dispute  to  have 
merit,  the  test  will  be  listed  in  the  VSR  under  the  category  "Tests  Under  Review."  The  test  result 
(pass  or  fail)  will  be  included  in  the  VSR  as  an  informational  test.  If  the  interpretation  finds  that 
the  test  is  valid,  the  test  will  be  reinstated  to  the  SGML  Test  Suite,  and  the  supplier  will  be 
informed  of  the  interpretation  results.  CSL  will  also  assign  a date  for  the  correction  of  the 
nonconformity,  typically  six  months  later  (to  facilitate  auditing  by  a subsequent  validation). 

Other  unresolved  disputes  will  be  listed  as  nonconformities  in  the  VSR.  If  a determination  is  later 
made  (in  favor  of  the  SGML  parser)  that  the  disputed  test  is  in  error,  the  VSR  will  be  corrected  and 
reissued  and,  if  necessary,  a Certificate  of  Validation  will  be  issued.  Any  appropriate  changes  will 
be  made  in  the  next  issue  of  the  VPL. 
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3.4  Funding  (Reimbursement) 

The  testing  laboratory  will  determine  the  fee  for  the  validation  service.  An  SGML  Validation 
Request  Form  will  be  submitted  to  the  requestor  approximately  90  calendar  days  prior  to  the  month 
of  the  scheduled  on-site  testing.  A signed  copy  of  the  SGML  Validation  Request  Form  together 
with  payment  for  the  total  cost  of  the  validation  must  be  returned  to  the  testing  laboratory  no  later 
than  60  calendar  days  prior  to  the  month  of  the  scheduled  on-site  testing.  No  work  will  be 
performed  until  there  is  assurance  that  funding  is  available  for  the  validation.  The  CSL 
administrative  fee  of  $1000.(X)  per  test  will  be  submitted  by  the  testing  laboratory  to  CSL  for  the 
evaluation  of  the  test  results.  This  fee  will  be  included  within  the  fee  the  testing  laboratory  charges 
for  the  validation  service. 


3.5  Release  of  Validation  Information 

Until  a VSR  is  registered,  requests  for  information  concerning  validation  of  an  SGML  parser  will 
be  directed  to  the  supplier. 

3.5.1  Publication  and  Proprietary  Data 

In  general  CSL  shall  have  the  right  to  use  all  information  gathered  in  the  course  of  developing  and 
administering  a conformance  testing  program  for  any  governmental  purpose,  but  shall  not  release 
such  information  publicly  except:  (1)  When  reporting  on  the  results  of  testing,  CSL  may  provide 
information,  subject  to  the  provisions  of  the  subparagraphs  below;  and  (2)  as  required  pursuant  to 
a request  under  the  Freedom  of  Information  Act  (5  U.S.C.  Section  552). 

3.5.2  Proprietary  Data 

The  supplier  of  information  shall  place  a Proprietary  notice  on  all  information  delivered  to  CSL  that 
the  supplier  asserts  is  proprietary.  Any  information  designated  as  proprietary  that  is  furnished  to 
CSL,  shall  be  used  by  CSL  only  for  the  purpose  of  carrying  out  validations.  Information  designated 
as  proprietary  shall  not  be  disclosed,  copied,  reproduced  or  otherwise  made  available  in  any  form 
to  any  other  person,  firm,  corporation,  partnership,  association,  or  other  entity  without  the  consent 
of  the  supplier  except  as  such  information  may  be  subject  to  disclosure  under  the  Freedom  of 
Information  Act  (5  U.S.C.  552).  CSL  will  use  its  best  efforts  to  protect  information  designated  as 
proprietary  from  unauthorized  disclosure.  CSL  will  not  be  liable  for  the  disclosure  of  information 
designated  as  proprietary  that,  after  notice  to  and  consultation  with  the  supplier,  CSL  determines 
may  not  lawfully  be  withheld  or  that  a court  of  competent  jurisdiction  requires  disclosed. 

3.5.3  Publication 

VSRs  registered  with  CSL  shall  be  made  available  to  the  public.  In  no  event,  however,  shall  the 
name  of  the  supplier  or  any  of  its  trademarks  and  trade  names  be  used  in  CSL  publications  without 
the  supplier’s  prior  written  consent.  With  respect  to  publication  in  the  VPL,  the  standard  Validation 
Customer  Agreement  shall  contain  the  supplier’s  written  consent. 
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CSL  and  the  supplier  shall  agree  to  confer  and  consult  prior  to  the  publication  of  data  to  assure  that 
no  Proprietary  Data  is  released  and  that  patent  rights  are  not  jeopardized.  Prior  to  publishing  a 
VSR,  the  supplier  shall  be  offered  an  opportunity  to  review  such  proposed  publication. 
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4.  SGML  PARSER  VALIDATION  SYSTEM 


The  Director  of  CSL,  or  designated  representative,  determines  the  SGML  Parser  Validation  System 
(SGML  Test  Suite)  version  or  release  to  be  used  for  validation. 

The  NIST  SGML  Test  Suite,  Version  1 .0,  becomes  the  official  version  of  the  SGML  Test  Suite  on 
January  1,  1995.  To  the  best  of  our  knowledge,  this  SGML  Test  Suite  evaluates  conformance  to 
the  minimum  requirements  of  FIPS  152. 

The  NIST  SGML  Test  Suite  may  be  licensed  from  CSL.  To  request  a copy  of  the  SGML  Test 
Suite  and/or  to  submit  questions  regarding  the  SGML  Test  Suite,  contact: 

National  Institute  of  Standards  and  Technology  (NIST) 

Computer  Systems  Laboratory 
Systems  and  Software  Technology  Division 
Office  Systems  Engineering  Group 
Building  225,  Room  B266 
Gaithersburg,  Maryland  20899  (U.S.A.) 

Ron  Wilson 

Telephone  (301)  975-3352  (Direct  line) 

(301)  975-3345  (Secretary) 

(301)  926-3696  (FAX) 

e-mail:  wilson@sst.ncsl.nist.gov  (INTERNET) 

4.1  SGML  Test  Suite  and  Release  Schedule 

A new  version  of  the  SGML  Test  Suite  will  be  issued  when  warranted  by  significant  changes. 
Changes  may  be  required  to  the  SGML  Test  Suite  to  correct  existing  errors,  enhance  test  routines, 
and  reflect  SGML  language  interpretations.  As  per  procedures  defined  within  Appendix  D, 
additional  tests  for  features  of  SGML  will  be  developed  and  added  to  the  SGML  Test  Suite.  Also, 
when  a new  version  of  the  SGML  standard  is  issued,  and  the  FIPS  is  revised,  the  SGML  Test  Suite 
will  normally  be  enhanced  to  test  for  the  new  features. 

When  a licensee  requests  SGML  parser  validation  on  either  a new  product  or  a different  platform, 
CSL  will  distribute  the  latest  version  of  the  validation  test  suite  for  an  additional  charge  equal  to 
twenty  percent  of  the  original  fee. 
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APPENDIX  A - NOTICE  OF  REGISTERED  VSR 


Notice  of  Registered  Validation  Summary  Report  (VSR) 

January  10,  1994 

The  information  below  will  be  listed  on  the  Computer  Systems  Laboratory  (CSL)  register  of  tested  SGML 
Parsers . 

The  SGML  parser  identified  in  this  Validation  Summary  Report  (VSR)  is  considered  to  be  "validated" 
because  it  did  pass  all  required  tests  for  a certificate  and  met  the  criteria  for  VSR  registration 
as  specified  in  the  SGML  Parser  Validation  Procedures.  The  VSR  should  be  reviewed  for  identification 
of  test  environment  parameters,  implementation-defined  specifications,  withdrawn  tests,  and  tests 
under  review. 


Report  Issued  to:  SGML-Vendor,  Inc. 

VSR  Number:  NIST-94/7011 

Vendor  Representative:  Ms.  Beatrice  Kim 

SGML  Parser  Identification:  S6ML-202,  Version  3.0  Pre-release 

Validation  Environment: 

SGML  Components:  S6ML-202  Engine,  Version  3.0  Pre-release 

Hardware:  CHIP486,  Model  1124 

Operating  System:  CHIP-UX,  Version  5.0.3 

Test  Date:  January  3,  1994 

FIPS  PUB:  152  SGML,  September  26,  1988 

Validation  System:  NIST  SGML  Test  Suite,  Version  1.0 

Approved  Changes  Dated:  December  2,  1993 
Tests  Under  Review:  None 

Features  and  Interfaces  Tested: 

Conformance  to  SGML  ISO  8879-1986 
Support  for  FIPS  152 


Signed:  

Roger  Martin 
Chief,  Division  872 
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APPENDIX  B - SGML  VALIDATION  REQUEST  FORM 


SGML  VALIDATION  REQUEST  FORM 
SGML  ( FIPS  152  ) 


Company  Name. 
Address  


1.  SGML  Software  Product  to  be  validated: 
Name/Version  # 


2.  Test  Environment:  Hardware  ID  Model  # 


Operating  System  ID/Version  # 


3.  Validation  Location:  

(if  different  from  above)  

4.  Validation  month:  Desired  month  Alternate  Month  

Deadline  (if  any)  for  completion  of  validation  

Validation  is  requested  for  a specific  RFP  (yes/no)?  

5.  Points  of  Contact: 

Contractual  Telephone  

Technical  Telephone  

This  is  not  a commitment  for  a NIST  validation.  The  undersigned  will  be 
contacted  by  NIST  to  discuss  procedures,  scheduling,  and  cost. 

Signature  Date  

(printed  name)  Phone  
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APPENDIX  C - TEST  METHOD  CONTROL  BOARD 


1.  TEST  METHOD  CONTROL  EXECUTIVE  COMMITTEE  (TMCEC) 

Based  on  the  document  ‘Test  Method  Control  Procedures  Model’  [4],  the  following  is  an  analysis 
of  how  such  a model  should  be  set  up  and  implemented  with  regard  to  the  day  to  day  running  and 
maintenance  of  the  Standard  Generalized  Markup  Language  (SGML)  validation  service  and  its 
software. 

Within  the  following  document  the  words  ’Test  Method’  refer  to  the  SGML  Test  System  and  it’s 
associated  documentation. 

1 . 1 Organization 

The  purpose  of  the  TMCEC  for  SGML  (hereinafter  referred  to  as  TMCECS)  is  to  resolve  issues 
in  a fair  and  speedy  fashion.  Communications  between  members  will  be  by  electronic  mail,  and 
in  cases  where  members  have  no  access  to  this  media,  facsimile  copy. 

1.1.1  Confirmation  of  Chairperson 

The  chairperson  shall  oversee  the  running  of  the  TMCECS  and  shall  be  the  designated 
representative  of  NIST.  The  chairperson  shall  be  the  designated  technical  authority  and  shall  have 
up  to  date  knowledge  of  the  status  of  the  service  and  suites  used  within  it. 

1.1.2  Designation  of  a Secretary 

The  role  of  secretary  in  the  TMCECS  shall  be  held  by,  or  appointed  by,  the  chairperson. 

1.1.3  Establishment  of  committee  voting  procedures 

The  voting  procedure  for  all  issues  brought  before  the  TMCECS  is  as  follows: 

An  issue  is  raised  with  any  member  of  the  TMCECS. 

The  issue  is  brought  to  the  attention  of  the  chairperson. 

The  chairperson  forwards  the  issue  to  the  members  of  the  TMCECS  (by  fax  or  e-mail)  for 
voting. 

The  members  of  the  TMCECS  shall  have  10  working  days  to  return  their  vote  on  the  issue 
in  question.  Votes  can  only  be  YES  or  NO.  A non-retumed  vote  is  counted  as  a YES. 

No  votes  will  be  reviewed  and  the  rationale  sent  to  the  TMCECS  members.  Members  will 
have  5 working  days  to  change  their  votes. 

A resolution  to  an  issue  in  question  is  accepted  after  75%  of  the  members  have  voted  in  it’s 
favor. 
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The  result  of  the  vote  is  made  known. 

1.1.4  Establishment  of  Membership 

The  membership  of  the  TMCECS  will  be  open  to  all  test  laboratories  offering  a SGML  test  service. 
The  offers  will  be  given  to  the  following  organizations  (see  Appendix  E). 

NIST 

EXOTERICA  CORPORATION 

It  may  be  decided  that  two  vendor  representatives  be  allowed  to  participate,  to  represent  the  client 
base  in  decisions  with  the  proviso  that  commercially  sensitive  material  will  be  sanitized  before 
being  discussed. 

1.1.5  Establishment  of  an  appropriate  Interface  Mechanism  with  the  Standards  Authority. 

The  interface  mechanism  with  the  standards  authority  may  be  the  designated  member  of  the 
TMCECS  who  also  is  a member  of  the  ISO  standards  committee. 

1 .2  Test  Method  Control  Executive  Committee 

The  committee  is  responsible  for: 

Resolving  challenges  to  the  test  method. 

Overseeing  the  maintenance  of  the  test  method. 

Coordinating  issues  of  interpretations  of  the  standard. 

Procedures  and  documentation  associated  with  the  test  method. 

Advising  on  the  proper  use  of  the  test  method. 

Recommending  changes  to  these  procedures. 

Initially  accepting/recognizing  new  test  methods. 

Recommending  improvements  for  the  next  revision  of  the  test  method. 

1.2.1.  Resolving  challenges  to  the  test  method. 

A situation  arising  in  which  a party  disagrees  with  the  way  that  the  SGML  validation  system  is 
maintained  in  either  the  actions  of  testing  or  in  the  test  suite  itself  should  be  dealt  with  using  the 
voting  procedures  described  in  Section  1.1.3  above. 

As  a result  of  submitting  a challenge  to  the  test  method  to  the  TMCECS,  one  of  five  resolutions 
may  be  made  as  indicated  in  Appendix  D,  ’Status  of  Tests.’ 
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1.2.2.  Overseeing  the  maintenance  of  the  test  method. 

NIST  will  oversee  the  maintenance  of  the  test  method.  As  changes  occur  to  the  ISO  Standard  and 
testing  experience  is  gained,  the  test  method  and  test  suite  may  require  updating.  The  TMCECS 
will  vote  on  recommended  changes  as  described  in  Section  1.1.3. 

1. 2.2.1  The  SGML  Test  Suite 

The  SGML  test  suite  is  updated  for  two  reasons: 

A major  change  is  made  to  the  existing  suite  through  the  addition  of  new  tests  against  new 
functionality. 

The  update  of  the  suite  is  to  incorporate  code  improvements  by  applying  bug  fixes  to  the 
existing  code. 

1.2. 2.2  SGML  Test  System  User  Guide 

At  the  time  of  every  update  to  the  SGML  test  software,  the  relevant  User  Guide  is  also  updated  to 
reflect  any  changes  that  have  been  made.  During  the  course  of  the  test  software’s  lifetime,  the  User 
Guide  may  be  found  to  contain  typographical  errors  or  inaccuracies.  These  are  dealt  with  by 
issuing  a page  update.  Each  page  of  the  User  Guide  carries  a date,  thus  enabling  the  User  to  know 
that  an  update  has  been  made  to  the  page  since  the  document  was  issued.  Changes  to  this 
document  should  not  need  to  go  before  the  TMCECS  unless  deemed  necessary  by  the  chairperson. 

1. 2.2.3  Test  Reports 

NIST  maintains  a SGML  Test  Report  Proforma.  This  report  is  supplied  to  all  test  labs.  It  is  a 

requirement  that  to  be  a test  laboratory,  the  laboratory  will  produce  a test  report  that  meets  NIST 
specifications.  All  test  reports  shall  contain  the  same  information  and  be  structured  in  the  same 
way.  In  order  to  ensure  that  this  is  the  case,  all  test  labs  should  forward  a sample  copy  of  the  final 
test  report  of  a validation  to  the  chairperson.  Issues  may  be  raised  regarding  the  content  of  the  test 
report. 

1. 2.2.4  Certificates 

Only  NIST  may  issue  Certificates  of  Validation. 

1.2.3.  Coordinating  issues  of  interpretations  of  the  standard. 

Disputes  based  on  interpretations  of  the  standard  shall  be  dealt  with  in  the  method  set  down  in 
Section  1.2.1. 

If  the  TMCECS  agree  that  an  interpretation  of  the  standard  is  required,  the  issue  is  sent  to  the 
appropriate  Standards  Committee  for  resolution  through  the  appointed  standards  committee  liaison 
organization. 

1.2.4.  Advising  Testing  Laboratories,  clients  and  others  on  the  proper  use  of  the  test  method. 

Each  Testing  Laboratory  is  responsible  for  advising  their  clients  on  usage  of  the  test  method. 
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Advice  to  other  test  labs  by  NIST  will  be  made  on  an  informal  basis,  unless  the  advice  may  effect 
all  labs  in  such  a way  that  the  procedures  may  be  affected. 

1.2.5.  Recommending  changes  to  these  procedures. 

The  function  of  the  TMCECS  shall  be  reviewed,  at  least,  once  a year.  The  chairperson  shall 
distribute  a questionnaire  in  which  each  member  will  be  asked  if  they  feel  any  changes  are  needed 
to  the  existing  practices. 

1 .2.6.  Initially  accepting/recognizing  new  test  methods. 

All  new  test  methods  shall  be  put  before  the  board  for  voting.  If  accepted,  the  methods  will  be 
implemented  at  the  first  opportunity.  A new  method,  if  accepted,  shall  also  be  recognized  by  those 
members  who  voted  against  the  issue. 

1.2.7.  Recommending  improvements  and  enhancements  for  the  next  revision  of  the  test  method. 

These  should  be  submitted  to  the  Secretary  for  review  by  the  TMCECS. 

1 .3  Parliamentary  Authority 

Robert’s  Rules  of  Order  or  other  established  alternatives  acceptable  to  the  Committee  shall  govern 
all  proceedings  of  meetings  not  addressed  herein. 
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2.  PROCEDURES 


Procedures  are  necessary  to  ensure  that  all  users  of  the  test  method  (Testing  Laboratories,  Clients,  etc.) 
use  the  same  test  method  at  all  times.  The  procedures  are  comprised  of  three  primary  areas: 

Test  method  maintenance; 

Disputes;  and 

Technical  assistance. 

2.1  Test  Method  Maintenance 

2.1.1  Test  Method  Releases 

2. 1.1.1  Types  of  Releases 

There  are  two  types  of  test  method  releases.  They  are  the  release  of  a new  version  of  the  test 
method  or  release  of  an  update  to  an  existing  version  of  the  test  method  through  bug  fixes.  Updates 
to  the  existing  version  of  the  test  method  shall  be  for  correcting  errors  in  the  test  method. 

2. 1.1. 2 Frequency  of  Test  Method  Release 

New  versions  of  the  test  method  should  not  be  released  more  frequently  than  once  a year.  New 
versions  of  the  test  method  shall  be  approved  by  the  Committee  prior  to  release. 

Updates  to  the  existing  version  of  the  test  method  are  issued  as  deemed  necessary  by  the 
Committee. 

2. 1.1. 3 Testing  of  New  Versions  of  the  Test  Method 

New  versions  of  the  test  method  are  made  available  for  field  testing  by  designated  beta  test  sites. 

2. 1.1. 4 Announcement  of  New  Releases  of  the  Test  Method 

An  announcement  will  be  made  when  new  releases  of  the  test  method  are  available.  Ordering 
information  and  the  effective  date  of  use  for  official  conformity  testing  will  be  included  in  this 
announcement. 

2. 1.1. 5 Use  of  New  Releases 

At  the  time  of  release  of  a new  version  of  the  test  method,  an  announcement  is  made  indicating 
when  it  replaces  the  current  version  for  formal  validation.  The  pre-testing  portion  of  the  conformity 
testing  process  must  be  performed  using  the  same  release  of  the  test  method  as  that  used  during  the 
on-site  portion  of  the  conformity  testing  process. 

2.1.2  Method  for  Changing  the  Test  Method 

Users  of  the  test  method  may  request  changes  to  any  portion  of  the  test  method;  e.g.,  test  suite, 
testing  support  tools,  or  associated  documentation.  Requests  for  change  shall  be  submitted  in 
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writing  or  electronic  mail  to  the  Secretary  of  the  Committee  or  the  testing  laboratory  conducting 
conformity  testing.  All  requests  received  by  a testing  laboratory  shall  be  forwarded  to  the 
Committee  for  resolution.  Requests  for  test  method  changes  should  be  submitted  using  the  format 
of  the  Software  Error  Report  provided  in  the  SGML  User  Guide. 

The  Committee  shall  evaluate  and  resolve  all  requests  for  change  of  the  test  method.  Changes  to 
the  test  method  shall  be  made  in  accordance  with  committee  approval  criteria  (see  Section  1.1.3). 
The  changes  shall  be  incorporated  into  the  test  method  either  in  the  bug  fix  or  next  release  of  the 
test  method. 

The  Committee  shall  authorize  the  withdrawal  of  any  test  case  which  is  found  to  be  incorrect.  In 
each  case,  the  test  case  is  withdrawn  from  the  current  version  of  the  Test  Method  used  for  all 
testing  performed  subsequent  to  the  date  of  withdrawal  of  the  test  case.  A list  of  withdrawn  test 
cases  will  be  maintained  by  the  Secretary.  The  list  of  withdrawn  test  cases  shall  be  included  in 
each  test  report.  Withdrawn  test  cases  do  not  count  towards  the  results  of  testing. 

2.1.3.  Impact  of  Interpretations  on  the  Test  Method 

The  TMCECS  shall  be  responsible  for  reviewing  all  interpretations  published  by  the  committee  of 
the  recognized  standards  body  responsible  for  the  development  of  the  standard  to  determine  any 
necessary  changes  to  the  test  method  resulting  from  the  interpretations. 

Any  necessary  changes  to  the  test  method  resulting  from  interpretations  of  the  Standard  language 
or  the  Test  Method  will  be  processed  in  accordance  with  these  procedures.  (See  Section  2.1.2, 
"Method  for  Changing  the  Test  Method") 

2.1.4.  Test  Method  Documentation 

Each  release  of  the  test  method  shall  be  accompanied  by  documentation  for  implementing  and  using 
the  test  method.  This  documentation  includes  User  Guide,  checksheets  and  instructions  for  the 
submission  of  Software  Error  Reports. 

2.1.5.  Changes  to  the  Test  Method  Control  Procedures 

Changes  to  these  procedures  may  be  recommended  and  submitted  to  the  Committee  for 
consideration. 

2.2  Resolution  of  Requests  for  Interpretations 

Requests  for  interpretation  should  be  submitted  to  the  Secretary  for  distribution  to  the  Committee 
membership  for  consideration  and  resolution.  Requests  for  interpretation  may  be  of  two  types: 

Interpretations  of  the  Standard  language. 

Interpretations  of  the  test  method. 

Any  test  cases  that  may  be  affected  by  pending  interpretations  shall  remain  in  the  Test  Method  (as 
Tests  Under  Review)  until  action  is  completed  on  the  interpretation.  Such  test  cases  may  result  in 
nonconformities  being  identified  in  the  test  report. 
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The  test  report  shall  state  that  an  interpretation  is  pending  in  these  cases.  Test  cases  subject  to 
interpretation  do  not  count  toward  the  results  of  testing. 

2.2.1.  Interpretations  of  Standard  Languages 

Requests  for  interpretation  concerning  interpretation  of  the  standard  language  specification  will  be 
submitted  to  the  appropriate  standards  organization  for  resolution.  Further  processing  of  the 
resulting  interpretation  will  be  in  accordance  with  these  procedures.  (See  Section  2.1.3,  "Impact 
of  Interpretations  on  the  Test  Method") 

2.2.2.  Challenges  to  the  Test  Method 

The  Committee  shall  evaluate  and  resolve  all  challenges  to  the  Test  Method.  These  resolutions 
shall  be  approved  in  accordance  with  the  committees’  established  rules  for  approval  (See  Section 
1.1.3).  Any  changes  to  the  Test  Method  resulting  from  the  resolution  of  these  challenges  will  be 
processed  in  accordance  with  these  procedures.  (See  Section  2.1.2,  "Method  for  Changing  the  Test 
Method") 

2.3  Technical  Assistance 

The  TMCECS  for  the  test  method  shall  provide  technical  assistance  to  the  Testing  Laboratories  and 
other  users  in  the  implementation  and  use  of  the  test  method.  The  Testing  Laboratories  shall 
provide  technical  assistance  to  users  of  the  test  method. 
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APPENDIX  D - STATUS  OF  TESTS 


STATUS  OF  TESTS 


The  following  Appendix  provides  information  of  the  status  of  programs  and  tests  of  the  SGML  test  suite 
if  they  are  submitted  to  the  dispute  process  of  the  TMCECS. 

Within  this  Appendix,  the  word  ’program’  or  ’audit  routine’  are  the  same.  A program  consists  of  a 
number  of  ’tests’. 

The  TMCECS  is  in  existence  to  solve  disputes  raised  by  parties  regarding  the  SGML  validation  suite  and 
the  procedures  that  exist  to  support  it.  The  major  use  of  the  TMCECS  will  be  to  solve  disputes  regarding 
the  SGML  suite  itself.  The  method  of  doing  this  will  always  be  the  same  - the  client  (or  test  laboratory) 
will  submit  a dispute  to  the  TMCECS  and  a decision  will  be  made  that  may  affect  the  status  of  the  test 
within  the  test  suite.  There  are  five  possible  results  of  doing  this. 

No  Change. 

The  TMCECS  do  not  agree  with  the  dispute  raised  and  decide  that  no  change  is  to  be  made. 

Bug  Fix  Issued. 

The  TMCECS  agree  with  the  dispute  raised  and  issue  a bug  fix  that  will  alter  the  audit  routine  in  such 
a way  that  the  number  of  tests  expected  to  be  attempted  remains  the  same.  Bug  fixes  falling  under  this 
category  will  generally  involve  the  correcting  of  syntactic  errors  in  the  code. 

Test  Withdrawn. 

The  TMCECS  agree  with  the  dispute  raised  and  withdraw  a test  from  an  audit  routine.  The  test  is 
withdrawn  since  the  change  required  to  be  made  to  correct  the  audit  routine  cannot  be  addressed  by  a bug 
fix. 

A test  is  withdrawn  using  a bug  fix  and  will  remain  withdrawn  until  the  next  release  of  the  test  suite. 
This  is  so  noted  in  the  test  report.  A decision  should  be  made  before  the  next  release  of  the  SGML  test 
system  as  to  the  final  status  of  the  test.  Where  possible,  the  problem  surrounding  the  test  should  be  dealt 
with  and  the  test  fixed  before  the  release  of  the  new  suite.  Where  this  is  not  possible  the  test  should  be 
given  the  final  status  of  ’Test  Deleted’. 

Test  Deleted. 

If  the  TMCECS  reach  the  conclusion  that  a test  of  the  SGML  test  suite  is  no  longer  suitable  for  inclusion 
in  the  suite  it  must  be  deleted.  During  the  life  cycle  of  the  suite  this  should  be  done  by  bug  fix  and  at 
the  time  of  the  new  release  the  code  connected  with  the  test  in  question  should  be  deleted  from  the  test 
suite.  The  expected  number  of  tests  to  be  attempted  for  that  audit  routine  should  be  altered  as  appropriate. 
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Test  Under  Review. 


The  TMCECS  classify  a test  as  TUR  if  they  decide  to  send  it  to  the  Standards  Interpretation  Committee. 
The  test  is  counted  as  TUR  until  the  response  from  the  Committee  is  known. 

For  the  purpose  of  validation,  a TUR  should  be  run  and  the  result  noted  along  with  the  information  that 
the  test  is  TUR. 
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APPENDIX  E - TMCECS  ORGANIZATION  REPRESENTATIVES 


TMCECS  ORGANIZATION  REPRESENTATIVE 

Representative 

Mailing  Address  email/Phone/FAX 

Ron  Wilson 

NIST  rwilson@nist.gov 

Computer  Systems  (301)975-3352 

Laboratory 

Building  225,  Room  B266 

Gaithersburg,  MD  20899 

USA 

John  McFadden 

Exoterica  Corporation  jrm@exoterica.com 

1545  Carling  Avenue  (613)722-1700 

Suite  404 

Ottawa 

Ontario,  Canada 

KIZ  8P9 
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